Metadata-free Audio-object Interactions

ABSTRACT

A method including, detecting interaction between the user and an audio-object via local tracking, determining an audio-object state modification based on the local tracking, and performing an audio-object interaction based on the audio-object state modification.

BACKGROUND Technical Field

The exemplary and non-limiting embodiments relate generally to renderingof free-viewpoint audio for presentation to a user using a spatialrendering engine.

Brief Description of Prior Developments

Free-viewpoint audio generally allows for a user to move around in theaudio (or generally, audio-visual or mediated reality) space andexperience the audio space in a manner that correctly corresponds to hislocation and orientation in it. This may enable various virtual reality(VR) and augmented reality (AR) use cases. The spatial audio mayconsist, for example, of a channel-based bed and audio-objects,audio-objects only, or any equivalent spatial audio representation.While moving in the space, the user may come into contact withaudio-objects, the user may distance themselves considerably from otherobjects, and new objects may also appear. The listening/rendering pointmay thereby adapt to the user's movement, and the user may interact withthe audio-objects, and/or the audio content may otherwise evolve due tothe changes relative to the rendering point or user action.

SUMMARY

The following summary is merely intended to be exemplary. The summary isnot intended to limit the scope of the claims.

In accordance with one aspect, an example method comprises, detectinginteraction between the user and an audio-object via local tracking,determining an audio-object state modification based on the localtracking, and sending the audio-object state modification to anaudio-object spatial rendering engine.

In accordance with another aspect, an example apparatus comprises atleast one processor; and at least one non-transitory memory includingcomputer program code, the at least one memory and the computer programcode configured to, with the at least one processor, cause the apparatusto: detect interaction between the user and an audio-object via localtracking, determine an audio-object state modification based on thelocal tracking, and send the audio-object state modification to anaudio-object spatial rendering engine.

In accordance with another aspect, an example apparatus comprises anon-transitory program storage device readable by a machine, tangiblyembodying a program of instructions executable by the machine forperforming operations, the operations comprising: detecting interactionbetween the user and an audio-object via local tracking, determining anaudio-object state modification based on the local tracking, and sendingthe audio-object state modification to an audio-object spatial renderingengine.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and other features are explained in the followingdescription, taken in connection with the accompanying drawings,wherein:

FIG. 1 is a diagram illustrating a reality system comprising features ofan example embodiment;

FIG. 2 is a diagram illustrating some components of the system shown inFIG. 1;

FIGS. 3a and 3b are diagrams illustrating characteristics offree-viewpoint content consumption;

FIG. 4 is an example illustration of a relationship between a userrendering position and an audio-object position;

FIG. 5 is an example illustration of a user rendering position againstaudio-object position over time;

FIG. 6 is another example illustration of a user rendering positionagainst audio-object position over time;

FIG. 7 is an example illustration of a user relationship to a localtracking area of an audio-object;

FIG. 8 is an illustration of interaction-area modification controlinstructions;

FIG. 9 is an example state machine illustration of audio-interactionevents;

FIG. 10 is an example diagram illustrating components of a renderingsystem;

FIG. 11 is an example illustration of a High-level block diagram formetadata-based audio-object interactions; and

FIG. 12 is an example illustration of a high-level block diagram of aswitched system implementing a reduced metadata rendering system as abackup system for a rendering system.

DETAILED DESCRIPTION OF EMBODIMENTS

Referring to FIG. 1, a diagram is shown illustrating a reality system100 incorporating features of an example embodiment. The reality system100 may be used by a user for augmented-reality (AR), virtual-reality(VR), or presence-captured (PC) experiences and content consumption, forexample, which incorporate free-viewpoint audio. Although the featureswill be described with reference to the example embodiments shown in thedrawings, it should be understood that features can be embodied in manyalternate forms of embodiments.

The system 100 generally comprises a visual system 110, an audio system120, a relative location system 130 and a reduced metadata (or ametadata free) rendering system 140. The visual system 110 is configuredto provide visual images to a user. For example, the visual system 12may comprise a virtual reality (VR) headset, goggles or glasses. Theaudio system 120 is configured to provide audio sound to the user, suchas by one or more speakers, a VR headset, or ear buds for example. Therelative location system 130 is configured to sense a location of theuser, such as the user's head for example, and determine the location ofthe user in the realm of the reality content consumption space. Themovement in the reality content consumption space may be based on actualuser movement, user-controlled movement, and/or some otherexternally-controlled movement or pre-determined movement, or anycombination of these. The user is able to move and turn their head inthe content consumption space of the free-viewpoint. The relativelocation system 130 may be able to change what the user sees and hearsbased upon the user's movement in the real-world; that real-worldmovement changing what the user sees and hears in the free-viewpointrendering.

The movement of the user, interaction with audio-objects and things seenand heard by the user may be defined by predetermined parametersincluding an effective distance parameter and a reversibility parameter.An effective distance parameter may be a core parameter that defines thedistance from which user interaction is considered for the currentaudio-object. In some embodiments, the effective distance parameter mayalso be considered a modification adjustment parameter, which may beapplied to modification of interactions, as described in U.S. patentapplication Ser. No. 15/293,607, filed Oct. 14, 2016, which is herebyincorporated by reference. A reversibility parameter may also beconsidered a core parameter, and may define the reversibility of theinteraction response. The reversibility parameter may also be considereda modification adjustment parameter. Although particular modes ofaudio-object interaction are described herein for ease of explanation,brevity and simplicity, it should be understood that the methodsdescribed herein may be applied to other types of audio-objectinteractions.

The user may be virtually located in the free-viewpoint content space,or in other words, receive a rendering corresponding to a location inthe free-viewpoint rendering. Audio-objects may be rendered to the userat this user location. The area around a selected listening point may bedefined based on user input, based on use case or content specificsettings, and/or based on particular implementations of the audiorendering. Additionally, the area may in some embodiments be defined atleast partly based on an indirect user or system setting such as theoverall output level of the system (for example, some sounds may not beaudible when the sound pressure level at the output is reduced). In suchinstances the output level input to an application may result inparticular sounds being not rendered because the sound level associatedwith these audio-objects may be considered imperceptible from thelistening point. In other instances, distant sounds with higher outputlevels (such as, for example, an explosion or similar loud event) may beexempted from the requirement (in other words, these sounds may berendered). A process such as dynamic range control may also affect therendering, and therefore the area, if the audio output level isconsidered in the area definition.

The reduced metadata rendering system 140 is configured to enablecontrolled audio-object interactions without needing transmission of anyassociated metadata. Thus, the method allows for a new alternativeimplementation of an audio-object interaction system. The reducedmetadata rendering system 140 may furthermore enable audio-objectinteractions in free-viewpoint audio experiences for such content thatdoes not include the metadata required by other rendering systems. Thereduced metadata rendering system 140 may implement rendering offree-viewpoint (or free-listening point; six-degrees-of-freedom; 6DoF,for example) audio for presentation to a user using a spatial renderingengine. In some implementations, reduced metadata rendering system 140may use an audio-object spatial modification engine or the spatialrendering engine may include functionality of an audio-object spatialmodification engine.

The reduced metadata rendering system 140 may implement processes forcontrolled audio-object interactions without needing transmission of anyassociated metadata, or metadata-free controlled audio-objectinteractions, based on a local tracking of user movement and activity.Specifically, reduced metadata rendering system 140 may track 1) adistance between the user and the audio-object to determine aninteraction area in which we consider audio-object interactions, and 2)a user movement relative to the audio-object (within the interactionarea) to determine transitions between interaction states.

The interaction states may each correspond to an interaction event (auser activity model and an audio-object interaction response). Theinteraction states may be defined by the implementer or derived, forexample, from an interaction event database. The transitions between thestates (or interaction events) may thereby be used to trigger eachseparate type of audio-object interaction response. The relevantresponses may differ between content, use case, and implementation. Noneof the responses depend on any transmitted metadata.

At least one distance related to initializing the local tracking may bedefined by the implementer or, for example, a content creator. In someembodiments, this distance may be derived automatically based, forexample, on past user behavior while consuming free-viewpoint audio.

Reduced metadata rendering system 140 may define the interaction areavia local tracking and thereby enable stabilization of the audio-objectrendering at a variable distance to the audio-object depending on realuser activity. In other words, the response of the reduced metadatarendering system 140 may be altered (for example, the response may beslightly different) each time, thereby improving the realism of theinteraction. The reduced metadata rendering system 140 may track theuser's local activity and further enable making of intuitive decisionson when to apply specific interaction rendering effects to the audiopresented to the user. Reduced metadata rendering system 140 mayimplement these steps together to significantly enhance the userexperience of free-viewpoint audio where no or only a reduced set ofmetadata is available.

Referring also to FIG. 2, the reality system 100 generally comprises oneor more controllers 210, one or more inputs 220 and one or more outputs230. The input(s) 220 may comprise, for example, location sensors of therelative location system 130 and the reduced metadata rendering system140, rendering information for reduced metadata rendering system 140,reality information from another device, such as over the Internet forexample, or any other suitable device for inputting information into thesystem 100. The output(s) 230 may comprise, for example, a display on aVR headset of the visual system 110, speakers of the audio system 120,and a communications output to communication information to anotherdevice. The controller(s) 210 may comprise one or more processors 240and one or more memory 250 having software 260 (or machine-readableinstructions)

Referring also to FIGS. 3a and 3b , diagrams 300, 350 illustratingcharacteristics of free-viewpoint content consumption are shown.

FIG. 3a illustrates a user 310 navigating around an audiovisualfree-viewpoint VR experience 300. The user 310 is surrounded by a naturescene, where the user 310 hears, for example, birds singing 320 aroundthe user 310 and bees buzzing 330 at some distance in front of the user310. As the user 310 moves forward (FIG. 3b ), the user 310 may comeinto contact with the beehive 340 that may, in terms of audio (oraudio-wise), consist, for example, of a single audio-object. This is anexample use case in which a definition for an interaction between theuser and the audio-object is required for an immersive free-viewpointaudio experience.

Existing systems for audio-object interactions in free-viewpoint audio(for example, systems as described in U.S. patent application Ser. No.15/293,607, and similar systems) may utilize metadata for defining howthe interactions should be detected and how they modify the rendering.For example, in FIG. 3b , the audio-object rendering may be modifiedbased on instructions derived from the metadata to amplify andaccelerate the buzzing of the bees in a circle around the user 310 andmake the sound of the bees 330 to follow the user 310 for a while evenif the user tried to leave this part of the scene. While this approachto modification may allow for well-defined and truly immersive userexperiences, there may also be problems arising from mobile use cases aswell as with regard to rendering of content that does not includemetadata (for example, legacy content for some systems).

In some instances, the metadata associated with the audio-objectinteractions may add a considerable amount of overhead to thetransmission. In instances of high-end VR applications the overheadassociated with audio-object interactions may not significantly affectperformance (especially if there is also high-quality video contentwhose bit rate typically dominates the consumption). However, ininstances of low bit rate AR audio solutions or while under severenetwork congestion, the overhead associated with audio-objectinteractions may make it difficult to apply these advanced features(especially if the audio-object interactions are dynamic and thusrequire frequent metadata updates). On the other hand, in otherinstances there may be (a considerable amount of) VR/AR content, orother audiovisual content that may be adapted for free-viewpoint VR/ARuse, that does not include these advanced metadata for audio-objectinteractions.

Reduced metadata rendering system 140 may implement a metadata-freesystem as an alternative and/or a backup for a system that includesadvanced metadata for audio-object interactions to better allowaudio-object interactions also for low bit rate AR scenarios and forimproved rendering of content (for example, legacy content) that doesnot have the required metadata.

Reduced metadata rendering system 140 may enable audio-objectinteractions without metadata based on instructions, which, from acontent creator's perspective, may appear to be arbitrary. However, theinteraction rendering cannot follow the content creator's instructionswithout any metadata indicating this. In instances of content that doesnot consider audio-object interactions (such as, for example, legacycontent) there may be no content creator input initially. Reducedmetadata rendering system 140 may provide an improved user experience inthese instances.

Referring also to FIG. 4, an example illustration 400 of a relationshipbetween a user rendering position 410 and an audio-object position 420based on a main trackable parameter 430 (in this instance, a distancebetween the user and audio-object) when no metadata related toaudio-object interactions is used is shown.

Referring also to FIG. 5, an example illustration 500 of a userrendering position 410 against an audio-object position 420 over time510 is shown. A distance of the user 410 from a middle line 550 of theaudio-object 420 is shown (530-L denotes a distance from the middle line550 in a left area while 530-R denotes the distance from the middle line550 in the right area). The user position (for example, in relation tothe audio-object position) is illustrated in a single dimension(left-to-right) with time 510 being illustrated on the vertical axis.

FIGS. 4 and 5, present illustrations showing the distance between theuser rendering position 410 and the audio-object position 420. Thisdistance may be tracked by reduced metadata rendering system 140.Reduced metadata rendering system 140 may thereby determine a portion ofthe information related to the overlap of the two positions before theactual overlap takes place and as the user 410 moves towards theaudio-object. FIG. 5 illustrates the user in FIG. 3a moving towards theaudio-object. In this case, reduced metadata rendering system 140 maytrack the distance 540 (of the user) along a single dimension (forexample, the left to right movement of FIG. 4) at several time instances520 (shown in FIG. 5 as 1, 2, 3, etc.).

In FIG. 5, the user 410 is at time instance 1 (shown as 520-1) at aconsiderable distance (shown as 540-1-L) from the audio-object. Thiscorresponds to the situation of FIG. 4. At time instance 520 2, the user410 has moved significantly closer to the audio-object. The audio-objectmay now appear very close to the user's head. The user 410 stops, butthere may still be small changes to the distance due to subtle movementsof the user 410. The user 410 may, for example, turn their head to viewthe scene, make small nods, correct their posture, or take a small stepin any direction. The audio-object 420 may thus end up oscillatingaround the user 410 along at least one dimension, as illustrated fortime instances 520 2-9 in FIG. 5 (shown as single numerals 2, 3 to 9, inFIG. 5). This may provide a very disturbing user experience when theaudio-object 420 is rendered to the user 410. Reduced metadata renderingsystem 140 may control the rendering such that it would appear morepleasant (for example, stable with smooth transitions) for the user 410.In addition to removing disturbances, reduced metadata rendering system140 may implement processes (for example, based on a second target ofcontrol for the audio-object interaction) to provide new information oran enhanced experience, for example as discussed above with respect toFIGS. 3a and 3 b.

FIG. 6 is an example illustration 600 of user rendering position againstaudio-object position over time. The position is illustrated in a singledimension (left-to-right) with time 510 being illustrated on thevertical axis.

As illustrated with respect to FIG. 6, reduced metadata rendering system140 may determine an additional area (for example, reduced metadatarendering system 140 may augment FIG. 5) by adding a region 610 (forexample, in place of the “audio-object center point line” 540, shown inFIG. 5) to better indicate that the user rendering position andaudio-object distances fall into two categories: the audio-object 420 iseither close to the rendering position (within region 610) or not closeto it (within area 620-L or 620-R, which may correspond to extendedareas outside of region 610). Reduced metadata rendering system 140 mayimplement processes to ensure that the user 410 response 1) does nothear blurry or oscillating audio-object position changes in therendering for time instances 520 2-9 and 12-15, and response 2) mayinstead hear a change in the audio-object rendering corresponding to aninteraction trigger. Reduced metadata rendering system 140 (or any audiorendering system) may require metadata to implement response 2) (forexample, without metadata response 2 may be difficult or impossible toexecute). Reduced metadata rendering system 140 may implement processesfor time instances 520 10-11 in FIG. 6, in which the user leaves and isoutside of the region 610.

According to an example, reduced metadata rendering system 140 mayimplement a distance tracking process for triggering and maintaining anaudio-object interaction. Reduced metadata rendering system 140 mayimplement the distance tracking process to calculate a distance betweenthe user rendering position and the audio-object position. This value(for example, the distance) and its change may be tracked over time.Reduced metadata rendering system 140 may thereby define whether theuser rendering position relative to the audio-object position is withinan area (for example region 610) where audio-object interaction may beconsidered.

Referring back to FIG. 6, reduced metadata rendering system 140 maydefine a value for the “size of the area” 610 that is to be considered“close” and correspondingly, areas that are “not so close”. The value(s)may be adaptive or dynamic. Reduced metadata rendering system 140 maydefine for each implementation or each general content type adistance/radius that corresponds to the correct size of region 610.Reduced metadata rendering system 140 may define the area in a dynamicfree-viewpoint audio use case in specific ways and additionally, oralternatively, reduced metadata rendering system 140 may trigger andcontrol an audio-object interaction using this area definition andrelated criteria.

Reduced metadata rendering system 140 may implement area definition toprovide a way for triggering and controlling audio-object interactions.Reduced metadata rendering system 140 may implement a dynamic areadefinition instances in which a static area definition is not optimal.Reduced metadata rendering system 140 may implement a static areadefinition for a simple stabilization of the rendering. In addition, dueto the nature of the user-on-audio-object overlaps and interaction inthe virtual space, reduced metadata rendering system 140 may center thearea 610 at positions other than the audio-object 420 although the area610 is to cover the audio-object 420. For example, in a particularinstance, the user 100 may be interacting with an audio-object 420 onone side of the audio-object 420, and then decide to move away, forexample, through said audio-object 420 (for example, on the other sideof said audio-object 420). If the area was centered at the audio-object420, the audio-object interaction would continue longer than required bythe corresponding (for example, real world based) logic of theimplementation. Reduced metadata rendering system 140 may thereforedefine a dynamic interaction area that may change at least one of itssize or its location based on the observed action of the user 410 inrelation to the audio-object 420 and the audio-object location. Reducedmetadata rendering system 140 may, in other words, track the localactivity of the user 410 relative to the audio-object 420 while the user410 is in the vicinity of the audio-object 420.

Referring also to FIG. 7, an example illustration of a user relationship700 to a local tracking area of an audio-object, is shown.

An initial stage 710, illustrates a user 410 approaching a localtracking area 610 of an audio-object 420, stage 720 illustrates the user410 entering the local tracking area 610 and triggering an audio-objectinteraction, and stage 730 illustrates the local tracking area 610 beingadjusted based on local user tracking.

Reduced metadata rendering system 140 may implement location trackingand area definition. FIG. 7 illustrates (different stages of) a userapproaching (and entering a local tracking area of) an audio-object 420.Reduced metadata rendering system 140 may specify animplementation-specific distance around the audio-object 420 where thelocal tracking is initially considered (and where reduced metadatarendering system 140 may also begin to consider the audio-objectinteraction). The distance may also be user-configurable, defined by thecontent creator (and loaded, for example, once per session) or, inadvanced embodiments, based on an ongoing process (for example, based ona learning algorithm) that accumulates user-specific data over time andthus allows automatic personalization of the experience.

The distance may correspond to a static area centered at theaudio-object 420, for example as shown at stage 710. As the user movescloser, he reaches the border of the tracked distance and triggers anaudio-object interaction, for example as shown at stage 720. We may thusbegin the local tracking when the user enters this pre-defined areaaround the audio-object. Alternatively, there may be different decisiondistances for the tracking and the actual interaction part. In thisexample, to simplify the description, the decision distances for thetracking and the actual interaction may be consider a same singledistance. The local tracking may be seen as defining a “center of mass”that is based on the user rendering position and the audio-objectposition, for example as shown at stage 730.

Referring also to FIG. 8 an illustration of interaction-areamodification control instructions is shown. These interaction-areamodification control instructions may be based on corresponding basicinstructions (corresponding to implementer defined logic or principles,such as based on real world scenarios) regarding the modification ofinteraction-areas in response to particular user motion.

Reduced metadata rendering system 140 may implement instructions basedon a model for controlling the dynamic interaction area (or center ofmass/CoM) 810. Reduced metadata rendering system 140 may implementinteraction-area tracking and modification based on core instructions orguidelines (for example, basic principles), such as shown in FIG. 8.With the user 410 entering the tracking distance 820 (shown at stage 1,block 815 in FIG. 8), the reduced metadata rendering system 140 mayinitialize CoM 810 at a point between the user position and theaudio-object 420. This may correspond to interaction area bounds 830,840 for the user 410 and audio-object 420. The exact location of the CoM810 may depend on the implementation and/or attributes such as theaudio-object size. The initial location may be, for example, the centerpoint between the user position and the audio-object position.

After initialization, reduced metadata rendering system 140 may beginthe local tracking. Step 2, block 825, of FIG. 8 illustrates the user410 approaching 850 the CoM 810, the position of which is maintained.

As the user 410 stops (or the absolute distance between the user 410 andthe audio-object 420 is otherwise maintained), the CoM 810 may movetowards the user 410 as seen in step 3, block 835. This movement maybecome slower the farther away from the audio-object the CoM 810 goes.At some point the CoM 810 may meet the user position, and reducedmetadata rendering system 140 may center the interaction area at theuser position 860. The interaction area may, in some embodiments, alsocover the actual audio-object position. Reduced metadata renderingsystem 140 may therefore render the audio-object 420 with the user 410(and not render the audio-object separately). Reduced metadata renderingsystem 140 may control the rendering via the interaction area.

Steps 4 a and 4 b (blocks 845 and 855) demonstrate two possibilitieswhere the user-to-audio-object distance is changed (after the userposition and CoM 810 have merged). In step 4 a, block 845, the user 410may move towards 870 the audio-object and the CoM 810 may follow theuser position.

In step 4 b, block 855, the user 410 may move away from the audio-objectposition. In this case, the CoM 810 may separate 886 from the userposition. The separation may, depending on the implementation (andvarious criteria such as the current distance to the audio-object)result in the CoM 810 being maintained or the CoM 810 following the userposition with a lag (inertia).

In some embodiments, the CoM 810 may move towards the audio-object 420if the user-to-audio-object distance becomes larger following theinitialization. Or, an increase of the user-to-audio-object distanceprior to merging of the user position and the CoM 810 may result in theCoM 810 moving towards the audio-object 420.

Reduced metadata rendering system 140 may define an interaction area forthe audio-object 420 that depends on the local tracking of the useractivity based on at least the user position (or analysis of the userposition). Reduced metadata rendering system 140 may also use otheraspects associated with the user or audio-object, such as speed of usermovement, past user movement, etc. In some embodiments, reduced metadatarendering system 140 may provide instructions for the CoM 810 to followthe user 410 while still providing a preference (or secondaryinstructions) for the CoM 810 to stay close to (or return to) theaudio-object 420.

FIG. 9 is an example illustration 900 of audio-interaction events thatmay be detected based on the local tracking of the user movement andbehavior in the proximity of an audio-object 420. Reduced metadatarendering system 140 may implement different states (for example, via astate machine, such as shown in FIG. 9) based on audio-interactionevents derived by local tracking.

Reduced metadata rendering system 140 may control rendering for audiointeractions without transmitted metadata. Reduced metadata renderingsystem 140 may analyze the particular audio interactions. When nometadata is available, reduced metadata rendering system 140 may derivepossible interaction effect by other means. Reduced metadata renderingsystem 140 may use analysis of local tracking of the user 410, such asmay already be performed for defining the interaction area 810. Asdescribed hereinabove, reduced metadata rendering system 140 may definethe interaction area based on the user position (distance). On the otherhand, reduced metadata rendering system 140 may determine theaudio-object interaction effect to be applied to the rendering based onother information derived from the local tracking of the user activity.As described above, the user 410 may, for example, consume/interact withan audio-object 420 firstly on one side of the audio-object 420 and thenproceed to leave this part of the scene, for example, through theaudio-object 420. Reduced metadata rendering system 140 may deteLiuinethat there are at least two parts in this activity. The first part maybe mostly a static consumption part, while the other part may be afaster movement (which, in this instance, may furthermore be through theaudio-object 420). Reduced metadata rendering system 140 may base themetadata-free interaction effect decisions on these types of user actiontransitions.

Referring again to FIG. 9, at 910, a first state in which the user 410is not interacting with an audio-object 420, is shown. The user 410 maythen enter a first interaction state with the audio-object at 920. Thisis denoted as an initial state, which may assume a first static userbehavior and trigger a first interaction response for the rendering ofthe audio-object 420. For example, the audio-object 420 may grow in sizeand be stabilized in rendering position relative to the user 410. Thesize may be, for example, relative to the user-to-audio-object distance.From this first static state we may then trigger new responses everytime a state is changed.

Reduced metadata rendering system 140 may have, for example, at least anon-static state 930 and/or a second static state differing from thefirst static state 920, where the user 410 has left the audio-objectinteraction area and then returned to interact with the sameaudio-object (‘return’) 940. For example, time instances 520 9-12 inFIG. 6 may correspond to an ending of, and subsequent relaunching of, anaudio-object interaction. The at least second interaction with the sameaudio-object may trigger a different response rendering.

In some embodiments, a user returning to interact with an audio objectafter a particular time has elapsed (for example, a minimum time haspassed) may trigger the initial state 920 instead of the ‘return’ state940. The reduced metadata rendering system 140 may therefore utilize atleast one rule, which may be based on time, for selecting between the atleast two static states that may be entered from the ‘no interaction’state 910. In further embodiments, the rule may be carried over from afirst user session to a second user session. It is understood that insome embodiments, only a single static state may be defined. Differentaudio objects may, depending on the implementation, have differentnumber of states.

The static state 920 may occur in instances in which there is usermovement (in addition to instances in which there is no user movement).For example, the static state may include instances in which there is arelatively local (on one side, in a certain segment, etc.) movement,and/or a relatively slow movement. Reduced metadata rendering system 140may thereby trigger a transition from a static state when at least auser movement distance is over a threshold and/or a user movement speedis over a threshold. Reduced metadata rendering system 140 may determinethat the interaction is entering a non-static state based on aparticular amount of said movement over a time period (a timethreshold). Transition from a non-static state to a static state mayrequire a more stable user activity than firstly remaining in a staticstate. The reduced metadata rendering system 140 may implement thetransition based on instructions provided by an implementer (forexample, based on a particular virtual environment, etc.) and, at leastin some cases, also based on the type of content.

The interaction states may each correspond to an interaction event(which may be a user activity model and a corresponding audio-objectinteraction response). These may be defined by the implementer or thecontent creator for the said implementation or content, respectively. Orthey may be derived, for example, from an interaction event database.

FIG. 10 is a diagram 1000 illustrating components of (corresponding tosteps in a process implementing) a reduced metadata rendering system140.

Reduced metadata rendering system 140 may, based on the local trackingand within the defined tracking area, determine when a user movementtriggers a state transition (FIG. 9) and start applying the audio-objectinteraction for the new state. In some example embodiments, reducedmetadata rendering system 140 may use a separate database for definingaudio-object interaction events. Reduced metadata rendering system 140may derive at least the user activity model for each state and/or theaudio-object interaction response for the each state from a database.

At block 1010, reduced metadata rendering system 140 may detect that auser is entering an audio-object vicinity (or distance). Reducedmetadata rendering system 140 may initialize local tracking at block1020 and perform local tracking and update of local tracking area atblock 1030. Reduced metadata rendering system 140 may update interactionevent states based on local tracking at block 1040. Alternatively, oradditionally, reduced metadata rendering system 140 may read interactionstate data from a database at block 1050. At block 1060, reducedmetadata rendering system 140 may perform audio-object statemodification based on a current interaction state. At block 1070,reduced metadata rendering system 140 may send modification informationto an audio object spatial rendering engine. Additionally, at block1080, reduced metadata rendering system 140 may, while user is in localtracking area, perform local tracking and update of local tracking area.

In alternative implementations, reduced metadata rendering system 140may utilize, for example, deep learning processes to further distinguishbetween various user activities or ways of reacting to audio-objects.Reduced metadata rendering system 140 may thereby allow forpersonalization of the system response. For example, reduced metadatarendering system 140 may utilize a training sequence of metadata-basedaudio-object interactions, where user movement is tracked, or reducedmetadata rendering system 140 may learn how user responds to defaultinteraction responses.

In further example embodiments, reduced metadata rendering system 140may analyze the audio-object 420 and the analysis result may affect atleast some of the audio-object interaction parameters, such as statesand thresholds. In further example embodiments, reduced metadatarendering system 140 may also analyze the physical rendering environment(user's room properties). Reduced metadata rendering system 140 maysimilarly affect the audio-object interaction rendering when nopre-defined metadata is used.

Referring now to FIG. 11, a high-level block diagram illustratingrendering for metadata-based audio-object interactions is shown.

Reduced metadata rendering system 140 may implement processes to providebackwards compatibility with previous systems, when compared tometadata-based systems, such as described in U.S. patent applicationSer. No. 15/293,607, which include metadata-based audio-objectinteractions. The metadata-based system may read the metadata (block1110), detect interaction (block 1120), and determine information for anaudio-object state modification (block 1130). The metadata-based systemmay then send the modification information to an audio-object spatialrendering engine (block 1140).

While reduced metadata rendering system 140 may allow controlled audiointeractions without additional metadata (as shown in FIG. 9), reducedmetadata rendering system 140 may also implement the metadata-freeprocesses in conjunction with systems, such as the metadata-based systemdescribed with respect to FIG. 11, which implement metadata-based audiointeractions. For example, in instances of network congestion, if a useris receiving a low-rate representation of a free-viewpoint audio scene,for example, over a wireless data link or communications system,complicated metadata may require a substantial amount of the overallbandwidth that would be better utilized for source-coding of the audiowaveform. Therefore, reduced metadata rendering system 140 may utilizethe metadata-free processes for rendering audio-object interactions andallow for the transmitter or a network element to drop the metadata andonly transmit the audio payload in the downlink.

Referring to FIG. 12, a high-level block diagram of a switched systemimplementing metadata-free processes for rendering audio-objectinteractions as a backup system for a system that uses metadata forrendering audio-object interactions (for example, a system such asdescribed in U.S. patent application Ser. No. 15/293,607) is shown.

As shown in FIG. 12, reduced metadata rendering system 140 may mirror(or perform as a substitute or alternate to) a metadata-based system,such as the system described with respect to FIG. 11, for instances inwhich metadata is not available. The functional blocks of FIG. 10 arethus re-arranged on the left-hand side of FIG. 12 to illustrate thehigh-level implementation of the two processes. The combined system maydetermine if metadata has been received at block 1210. If metadata hasnot been received, reduced metadata rendering system 140 may detectinteraction via local tracking at block 1220 and determine anaudio-object state modification based on local tracking at block 1230.Reduced metadata rendering system 140 may then send modificationinformation to audio-object spatial rendering engine at block 1140.However, if metadata has been received, metadata-based system mayperform steps 1110 to 1140, as described herein above with respect toFIG. 11.

Reduced metadata rendering system 140 may implement processes, forexample, under heavy network congestion when metadata transmission mayneed to be dropped to save bandwidth or to allocate it in a way that isperceptually more beneficial. Thus, an administrator (or a combinedsystem, such as described with respect to FIG. 12) may run themetadata-based system when metadata is received and switch to thereduced metadata rendering system 140 when no metadata is available.However, in these instances the combined system may be required tointerpolate the effects in order not to create discontinuities if theswitching between the two modes (branches) is frequent. For example,reduced metadata rendering system 140 may determine a metadata-freerendering utilizing the information about the past states derived fromthe received metadata in previous frames. The combined system may have adefault setting so that as long as data rate is sufficient, the combined(or switched) system processes the rendering for the audio-objectinteractions via the metadata-based system (and only uses the reducedmetadata rendering system 140 when no metadata is available).

Note that although the preceding implementations are described withrespect to user movement, dynamic audio-objects may also movethemselves, which may also affect the distance between the user positionand the audio-object. In instance of determination ofuser-to-audio-object distance, the relative distance is measured, andreduced metadata rendering system 140 may discount whether the movementis due to the user moving or the audio-object moving. However, ininstances of using local tracking of the user activity for determiningthe audio-interaction effects, the actual user movement is of interest.If the audio-object is also moving, reduced metadata rendering system140 may compensate for the tracking in at least some embodiments.

The metadata-free rendering of audio-object interactions may providetechnical advantages and/or enhance the end-user experience. At a highlevel, the processes may enable a stable audio-object rendering underaudio-object interaction with no or reduced set of metadata available atthe renderer. These processes are thereby suitable for example for verylow bit rate VR systems where metadata transmission may not be favoredand free-viewpoint rendering of legacy content that is not supported bythe full-metadata system.

Reduced metadata rendering system 140 may thereby make it possible forthe user to experience free-viewpoint audio based on both legacy contentand new VR-specific content (for example, content that includes metadatafor audio-object interactions). The interaction-area tracking may enablestabilizing the audio-object rendering. The tracking of user's localactivity may further enable making further decisions on when to applyspecific interaction rendering effects to the audio. Together thesesteps may significantly enhance the user experience. While reducedmetadata rendering system 140 may not follow content creator orimplementer instructions when no metadata is available, reduced metadatarendering system 140 may, in some instances, provide options (or makedecisions) that correspond for example to typical content creatordecisions at the renderer. Reduced metadata rendering system 140 mayalso implement personalization of audio-object interactions based on themodeling of a specific user's typical interaction style whileexperiencing free-viewpoint audio.

One advantage of the metadata-free rendering of audio-objectinteractions described herein is that it can be implemented as astand-alone system (thus, it offers a new, alternative implementationfor free-viewpoint audio-object interaction rendering) and in additionas a backup system for metadata-based systems (thus, improving theexisting system for lower bit rates and legacy content). It can thus beused independently from metadata-based systems (in terms of not needingmetadata) or in conjunction with metadata-based systems (when metadatais offered but it is not available due to transmission issues).

In accordance with an example, a method may include detectinginteraction between the user and an audio-object via local tracking,determining an audio-object state modification based on the localtracking, and sending the audio-object state modification to anaudio-object spatial rendering engine. The method may also includeperforming an audio-object interaction based on the audio-object statemodification.

In accordance with another example, an example apparatus may comprise atleast one processor; and at least one non-transitory memory includingcomputer program code, the at least one memory and the computer programcode configured to, with the at least one processor, cause the apparatusto: determine whether metadata associated with a user for audio-objectinteractions has been received, detect interaction between the user andan audio-object via local tracking in response to determination thatmetadata associated with the user for audio-object interactions has notbeen received, determine an audio-object state modification based on thelocal tracking, and send the audio-object state modification to anaudio-object spatial rendering engine.

In accordance with another example, an example apparatus may comprise anon-transitory program storage device readable by a machine, tangiblyembodying a program of instructions executable by the machine forperforming operations, the operations comprising: detecting interactionbetween the user and an audio-object via local tracking, determining anaudio-object state modification based on the local tracking, and sendingthe audio-object state modification to an audio-object spatial renderingengine.

In accordance with another example, an example apparatus comprises:means for determining whether metadata associated with a user foraudio-object interactions has been received, means for detectinginteraction between the user and an audio-object via local tracking inresponse to determination that metadata associated with the user foraudio-object interactions has not been received, means for determiningan audio-object state modification based on the local tracking, andmeans for sending the audio-object state modification to an audio-objectspatial rendering engine.

Any combination of one or more computer readable medium(s) may beutilized as the memory. The computer readable medium may be a computerreadable signal medium or a non-transitory computer readable storagemedium. A non-transitory computer readable storage medium does notinclude propagating signals and may be, for example, but not limited to,an electronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, or device, or any suitable combinationof the foregoing. More specific examples (a non-exhaustive list) of thecomputer readable storage medium would include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing.

It should be understood that the foregoing description is onlyillustrative. Various alternatives and modifications can be devised bythose skilled in the art. For example, features recited in the variousdependent claims could be combined with each other in any suitablecombination(s). In addition, features from different embodimentsdescribed above could be selectively combined into a new embodiment.Accordingly, the description is intended to embrace all suchalternatives, modifications and variances which fall within the scope ofthe appended claims.

1-20. (canceled)
 21. A method comprising: determining an amount ofreceived metadata; enabling detection of interaction between a user andan audio-object at an interaction area based, at least partially, on thedetermined amount of received metadata, wherein the detecting of theinteraction between the user and the audio-object is based on thereceived metadata when the determined amount of the received metadatacomprises a first amount, wherein the detecting of the interactionbetween the user and the audio-object is based on local tracking whenthe determined amount of the received metadata comprises a secondamount, where the first amount is different from the second amount;determining a state of the audio-object based, at least partially, on adetermination of whether the interaction is detected, wherein thedetermined state is one of a plurality of audio-object states; andrendering the audio-object based on the determined state of theaudio-object.
 22. The method of claim 21, wherein one or more states areassociated with the first amount, wherein one state is associated withthe second amount.
 23. The method of claim 21, wherein the first amountis not zero, wherein the second amount is zero.
 24. The method of claim21, further comprising: determining an audio-object state modificationbased on the detected interaction; transitioning between a firstinteraction state and a second interaction state based on the determinedaudio-object state modification; and performing an audio-objectinteraction response based on the transitioning.
 25. The method of claim21, wherein the detecting of the interaction between the user and theaudio-object based on local tracking further comprises: detecting theuser entering a vicinity of the audio-object; initializing the localtracking; and performing the local tracking and updating a localtracking area.
 26. The method of claim 21, further comprising:determining the interaction area based on a center of mass associatedwith the user and a center of mass associated with the audio-object. 27.An apparatus comprising: at least one processor; and at least onenon-transitory memory including computer program code; the at least onememory and the computer program code configured to, with the at leastone processor, cause the apparatus at least to perform: determine anamount of received metadata; enable detection of interaction between auser and an audio-object at an interaction area based, at leastpartially, on the determined amount of received metadata, whereindetecting the interaction between the user and the audio-object is basedon the received metadata when the determined amount of the receivedmetadata comprises a first amount, wherein detecting the interactionbetween the user and the audio-object is based on local tracking whenthe determined amount of the received metadata comprises a secondamount, where the first amount is different from the second amount;determine a state of the audio-object based, at least partially, on adetermination of whether the interaction is detected, wherein thedetermined state is one of a plurality of audio-object states; and causerendering of the audio-object based on the determined state of theaudio-object.
 28. The apparatus of claim 27, wherein one or more statesare associated with the first amount, wherein one state is associatedwith the second amount.
 29. The apparatus of claim 27, wherein the firstamount is not zero, wherein the second amount is zero.
 30. The apparatusof claim 27, wherein the at least one memory and the computer programcode further configured to, with the at least one processor, cause theapparatus at least to perform: determine an audio-object statemodification based on the detected interaction; transition between afirst interaction state and a second interaction state based on thedetermined audio-object state modification; and perform an audio-objectinteraction response based on the transitioning.
 31. The apparatus ofclaim 27, wherein detecting the interaction between the user and theaudio-object based on local tracking further comprises the at least onememory and the computer program code are configured to, with the atleast one processor, cause the apparatus at least to: detect the userentering a vicinity of the audio-object; initialize the local tracking;and perform the local tracking and update a local tracking area.
 32. Theapparatus of claim 27, wherein the at least one memory and the computerprogram code further configured to, with the at least one processor,cause the apparatus at least to perform: determine the interaction areabased on a center of mass associated with the user and a center of massassociated with the audio-object.
 33. A non-transitory computer-readablemedium comprising program instructions stored thereon which, whenexecuted with at least one processor, cause the at least one processorto: determine an amount of received metadata; enable detection ofinteraction between a user and an audio-object at an interaction areabased, at least partially, on the determined amount of receivedmetadata, wherein detecting the interaction between the user and theaudio-object is based on the received metadata when the determinedamount of the received metadata comprises a first amount, whereindetecting the interaction between the user and the audio-object is basedon local tracking when the determined amount of the received metadatacomprises a second amount, where the first amount is different from thesecond amount; determine a state of the audio-object based, at leastpartially, on a determination of whether the interaction is detected,wherein the determined state is one of a plurality of audio-objectstates; and cause rendering of the audio-object based on the determinedstate of the audio-object.
 34. The non-transitory computer-readablemedium of claim 33, wherein one or more states are associated with thefirst amount, wherein one state is associated with the second amount.35. The non-transitory computer-readable medium of claim 33, wherein thefirst amount is not zero, wherein the second amount is zero.
 36. Thenon-transitory computer-readable medium of claim 33, wherein the programinstructions are further configured to, when executed with at least oneprocessor, cause the at least one processor to: determine anaudio-object state modification based on the detected interaction;transition between a first interaction state and a second interactionstate based on the determined audio-object state modification; andperform an audio-object interaction response based on the transitioning.37. The non-transitory computer-readable medium of claim 33, whereindetecting the interaction between the user and the audio-object based onlocal tracking further comprises the program instructions are configuredto, when executed with at least one processor, cause the at least oneprocessor to: detect the user entering a vicinity of the audio-object;initialize the local tracking; and perform the local tracking and updatea local tracking area.
 38. The non-transitory computer-readable mediumof claim 33, wherein the program instructions are further configured to,when executed with at least one processor, cause the at least oneprocessor to: determine the interaction area based on a center of massassociated with the user and a center of mass associated with theaudio-object.